IPython.parallel doesn't do much in the way of serialization. It has custom zero-copy handling of numpy arrays, but other than that, it doesn't do anything other than the bare minimum to make basic interactively defined functions and classes sendable.
There are a few projects that extend pickle to make just about anything sendable, and one of these is dill.
To install dill:
pip install --pre dillFirst, as always, we create a task function, this time with a closure
In [1]:
def make_closure(a):
"""make a function with a closure, and return it"""
def has_closure(b):
return a * b
return has_closure
In [2]:
closed = make_closure(5)
In [3]:
closed(2)
Out[3]:
In [4]:
import pickle
Without help, pickle can't deal with closures
In [5]:
pickle.dumps(closed)
But after we import dill, magic happens
In [6]:
import dill
In [7]:
pickle.dumps(closed)[:64] + '...'
Out[7]:
So from now on, pretty much everything is pickleable.
As usual, we start by creating our Client and View
In [8]:
from IPython import parallel
rc = parallel.Client()
view = rc.load_balanced_view()
Now let's try sending our function with a closure:
In [9]:
view.apply_sync(closed, 3)
Oops, no dice. For IPython to work with dill,
there are one or two more steps. IPython will do these for you if you call DirectView.use_dill:
In [10]:
rc[:].use_dill()
Out[10]:
This is equivalent to
from IPython.utils.pickleutil import use_dill
use_dill()
rc[:].apply(use_dill)
Now let's try again
In [11]:
view.apply_sync(closed, 3)
Out[11]:
Yay! Now we can use dill to allow IPython.parallel to send anything.
And that's it! We can send closures and other previously non-pickleables to our engines.
Let's give it a try now:
In [12]:
remote_closure = view.apply_sync(make_closure, 4)
remote_closure(5)
Out[12]:
But wait, there's more!
At this point, we can send/recv all kinds of stuff
In [13]:
def outer(a):
def inner(b):
def inner_again(c):
return c * b * a
return inner_again
return inner
So outer returns a function with a closure, which returns a function with a closure.
Now, we can resolve the first closure on the engine, the second here, and the third on a different engine, after passing through a lambda we define here and call there, just for good measure.
In [14]:
view.apply_sync(lambda f: f(3),view.apply_sync(outer, 1)(2))
Out[14]:
And for good measure, let's test that normal execution still works:
In [15]:
%px foo = 5
print(rc[:]['foo'])
rc[:]['bar'] = lambda : 2 * foo
rc[:].apply_sync(parallel.Reference('bar'))
Out[15]:
And test that the @interactive decorator works
In [16]:
%%file testdill.py
from IPython.parallel import interactive
@interactive
class C(object):
a = 5
@interactive
class D(C):
b = 10
@interactive
def foo(a):
return a * b
In [17]:
import testdill
In [18]:
v = rc[-1]
v['D'] = testdill.D
d = v.apply_sync(lambda : D())
print d.a, d.b
In [19]:
v['b'] = 10
v.apply_sync(testdill.foo, 5)
Out[19]: